Capstone Project¶

II. Analysis¶

Data Exploration¶

The market data is provided by Quantopian. According to their contract with the original data vendors, this marked data can only be obtained and used within the Quantopian research environment. The API-function get_pricing returns open-high-low-close (OHLC) data for a large number of assets, with a maximum resolution of 1 minute over several years. The data is split- but not divident adjusted. This does however not pose any problem, since the agent is not compared to a market benchmark. In this project, SPY close data with a resolution of 1 minute is used, for a period of 1 year(+).

Exploratory Visualization¶

The market data is obtained through the get_pricing in the Quantopian research environment. This function returns a pandas.Panel-object with columns depending on the requested data and a datetime-index. For more details, see https://www.quantopian.com/help.

The following code snippet imports the historical data of the S&P 500-etf (SPY) which is used in all subsequent parts of this project:

The price data is used to calculate the market state, which is based on the values of three popular technical indicators. The following code snippet calculates the MACD, RSI and Bollinger Bands with a minute resolution for one specific day, using the talib implementation (see https://mrjbq7.github.io/ta-lib/). The values of the technical indicators and the SPY price are then combined in a single pandas.DataFrame, to pass them to a plotting function.

Please note that the first several indicator values are automatically set to NaN. This is an expected result, since the indicators are based on moving averages that need a certain period of historical data for its calculation. It is also a desired result, since the plotting functions just ignore NaN, which is the correct treatment of for non-existing values.

To visualize the complete data, the following function is defined:

To illustrate the market data, the DataFrame constructed above is passed to the plot_data function:

Algorithms and Techniques¶

The self-learning trading agent uses the Q-learning algorithm to learn making trading decisions, which - in its general form - can be stated as follows:

                 Initialize Q(s, a), ∀s ∈ S, a ∈ A(s), arbitrarily, and Q(terminal-state, ·) = 0 

                 Repeat (for each episode): 

                   Initialize S

                   Repeat (for each step of episode): 

                      Choose A from S using policy derived from Q (e.g., ε-greedy) 

                      Take action A, observe R, S' 

                      Q(S, A) ← Q(S, A) + α[R + γ*max<a>Q(S', a) − Q(S, A)]  

                      S ← S'  

                   until S is terminal 

Where

• S is the set of possible states a learner can experience in a given environment
• A(s) is the set of actions a learner can choose from in a specific state s
• Q(s, a) maps each state/action pair to a numeric value representing the reward an agent can expect, if he selects action a in state s
• An episode is one repetition of the same learning task, starting in a start state and ending in a terminal state
• ε-greedy is a method to control the exploration/exploitation tradeoff
• R is the reward the agent receives for taking action a in state s
• α is the learning rate which can be set to any value between 0 and 1 and controls the step size in the direction of the gradient on any update step. a lower value for α requires more update steps to find an optimum, but avoids getting stuck in local maxima or minima.
• γ is the discount rate, which controls the tradeoff between immediate reward and longer term gratification. The value of γ can be set to any value between 0 and 1, where a value of 0 results in a learner focusing only on the immediate reward for the current step, while a larger value means the learner also considers later rewards for taking a specific action a in a specific state s.

Several aspects of the self-learning trader system can easily be assigned to individual elements of the algorithm:

• An episode equals one trading day, the end of an episode is not marked by a terminal state, but by the end of a trading session
• The set of states S are derived from values and specifics of the technical indicators introduced above
• The set of possible actions A(s) in state s is defined by position sub-state
• The reward for an existing position is calculated by (current_price-entry_price)sign(position_size) at every update step, where position_size can either be 100,-100 depending on whether a long, a short or no position was taken. If there is no current position, the reward is 0.
• The step size is 1 minute, i.e. the update part of the algorithm is executed every minute for each episode

The general algorithm was slightly adapted due to some specifics of the problem domain and restriction of the environment. The version implemented for this project can be stated as follows:

                 Repeat (for each trading day):

                   prevS<-None  

                   prevA<-None

                   Repeat (for each trading_day in trading_days):

                      R <- current reward (i.e, value of current position)

                      S <- current state

                      If S not in Q-table, Initialize Q(S, a), ∀s a ∈ A(s)

                      Choose A from S using policy derived from Q (ε-greedy) 

                      Take action A 

                      if prevS is not None and prevA is not None

                          Q(prevS,prevA) ← Q(prevS,prevA) + α[R + γ*max<a>Q(S, a) − Q(prevS, prevA)]  

                      prevS<-S  

                      prevA<-A 

                   Until 30 minutes before trading session end

α will be set to a low value, to prevent getting stuck in local minima. To encourage the agent to favour longer term trades dending on indicator trend signals, γ is set to 0.8. At the same time, the high value of γ is supposed to prevent short term trades that are highly susceptible to noise. epsilon is set to 0.1, such that - in frequentist terms - the agent uniformly draws from the set of allowed actions at every ninth update step, instead of taking the action a which maximizes Q(S,a). This ensures that the agent keeps exploring even in later episodes, which intuitively seems desirable given the complex and dynamic characteristics of the market environment.

Benchmark¶

Main purpose of this project is to answer the question, if a Q-learning trading agent is able to learn anything useful at all, given a (simplified) day-trading setting. The performance of the the learning agent is therefore not compared to a common market benchmark, but much rather to another agent - a "monkey trader" - which definitely does not learn anything at all. This monkey trader acts in exactly the same environment and follows the same state transition rules as the self-learning agent. The only difference is, that the monkey trader at any update step just uniformly draws from the set of allowed actions and thus does not follow any learned policy whatsoever. If the self-learning agent is not able to perform significantly better than the monkey trader, it is not worth pursuing further. If it does perform considerably better, the next step would be to compare its performance to a common market benchmark, to see if it could be a viable candidate for an automated, active trading system.

III. Methodology¶

Data Preprocessing¶

The data exhibits no abnormalities, and can be used without preprocessing as returned by the get_pricing function.

Agent implementation¶

Using the base abstraction, the MonkeyTrader-benchmark only has to override the choose_next_action method, such that an action is uniformly drawn from the set of allowed actions at every update step:

The QLearningTrader uses the base abstraction and implements/overides all methods required for Q-learning:

Environment implementation¶

The Quantopian research environment is still in a beta phase, i.e. it may contain some bugs and will very likely change before release. These restrictions of the Quantopian research environment are however more than outweighed by the following advantages:

• Professional, split-adjusted market data, for a large number of assets, over long time periods and with minute resolution
• Order infrastructure, that allows to place orders under (simulated) market conditions including slippage and commision models
• Portfolio infrastructure which allows the analysis of current positions
• Comprehensive performance measure facilities, including the calculation of most common portfolio metrics
• Almost seamless portability of algorithms developed in the research environment to a live trading environment (Quantopian IDE), which supports paper- and live trading against a real broker account
• Easy collaboration opportunities, enabled by the notebook hosting

The first method is executed thirty minutes before session and activates a signal that triggers the agent to close any open position at the next update step:

The following initialize functions are called once at the beginning of the execution of each trading algorithm instance. Every algorithm-global variable has to be initialized and appended to the context-object in this function. Two versions of this function are needed, one for the MonkeyTrader benchmark and one for the QLearningTrader.

The create_state function below derives state information from the current values and characteristics of the 3 indicators at each update step. The state is defined based on some intuition and even less trading experience of the author. This aspect may be a good target for experimentation. The only restriction is, that the state has to be discrete for Q-table based learning. With continous states, the state space would blow up and the curse of dimensionality would very likely smash any performance advantage of the Q-learning trader.

The following before_trading_start function is called each day before the trading session starts and is used to initialize and update some variables on a daily basis.

The handle_data function below is called every minute during each trading day by the Quantopian trading algorithm machinery. In this project, handle_data is used to get the most recent prices, calculate the current indicator values, create the state and call the update function of the trading agent. Furthermore, relevant data is recorded for later analysis.

Refinement¶

The current notebook describes the first iteration of the QLearningTrader implementation. No refinements can be reported, therefore.

IV. Results¶

Model Evaluation and Validation¶

To illustrate the trading decisions of the MonkeyTrader and the QLearning trader, the analyze method below first extracts the transaction objects and then passes the recorded market and indicator data to the plot_data function, along with the transactions objects. To limit the number of plots, only the first and the last 3 trading days are plotted for each trader.

The following code snippets combine all the elements implemented above in two TradingAlgorithm objects, which can be run by the Quantopian backtest enviromnent. The monkey trader runs first:

And now the QLearningTrader:

The individual trades are represented by dashed lines in the plots above, where red indicates a losing and and green a winning trade.

Finally, the total_rewards of both traders are plottet in a single graph, to compare their performance directly:

The direct comparison between the two traders reveals, that the q-learning trader obviously learns something and achieves a significantly higher total reward than the monkey trader benchmark. However, the absolute performance is by no means satisfying. And a review of the 3 final trading days shows, that the q-learning trader favours extremely short trades, contrary to the initial assumption that a higher γ would lead to longer term trades.

Justification¶

A day trading setting such as the one used in this project lends itself nicely to a Q-learning implementation: the episodes are naturally defined by the individual days, state and reward can easily be derived from an existing market environment and the actions are limited to a small set of trade types. Due to the large number of possible combinations of the individual elements and the lack of easily reproducible benchmarks however, it is hard to justify and compare the results. The QLearningTrader implemented in this project, represents one possible approach to applying QLearning to day trading and compared to the constructed benchmark, the results really are significantly better. Although the absolute results may not justify the use of the QLearningTrader as day trading system, the implementation at the very least provides ample opportunity for further experimentation and research.

V. Conclusion¶

Free-Form Visualization¶

The Q-table at the core of any Q-Learning algorithm maps states and actions to a real number, which represents the value (or expected reward) for taking action a in state s. In the case of the QLearningTrader, Q(s,a) thus returns a value measuring the expected win or loss for taking trading action a in market state s. Upon completion of the learning algorithm, the Q-table contains the learned policy of the QLearningTrader, since in every state it takes the trading action with the max value for that particular state, i.e. argmax[a]Q(s,a). A more detailed inspection of the q-table is therefore a good approach to gain a deeper insight into what the QLearningTrader has actually learned.

For a general graphic overview, the Q-table is

The heatmap above reveals several interesting points. The most notable finding is that the values for the close action zero in any state. This points to a major flaw in the reward system of the QLearningTrader: At every update step the reward is calculated based on the positions resulting from the previous update step, the reward for close will always be zero.

Another interesting point is, that only the none actions have positive values, while the values of the two trading actions are <=0 in all states. That means, the QLearningTrader was not able to find a highly provitable trading action in any state. This finding is related to the more general one, that the q-values are altogether very low, which indicates that the QLearningTrader wasn't able to find any strong signal at all.

A finding that actually confirms some initial design decisions is the relatively small and therefore tractable state space. This could be achieved by deriving a set of discrete states from indicator signals and by limiting the set of trading actions.

Reflection¶

The task of developing a reinforcement learning trading agent posed several interesting challenges.

The first difficulty was to find related work, that could be used for benchmarking and/or orientation. Several papers exist applying different approaches of reinforcement learning to the trading and investment domain. However, the solutions and algorithms introduced by these papers vary widely and are often rather involved and difficult to reproduce. The design and implementation for this project was therefore largely developed from scratch, using some code snippets from a previous project.

Another difficult task was to find a good market environment with sufficient and reliable market data and with a simulation and backtesting infrastructure to run the Q-learning algorithm over a large number of episodes. Although the Quantopian research environment proved to be a very good choice in this regard, several aspects had to be adapted and extended, such as the plotting functionality to illustrate and analyze the trades.

Given theses challenges and considering the fact that the use of reinforcement learning in investment and trading is still a relatively new research area, the results of the simplified approach introduced in this project are quite respectable. Not only could it be shown that the QLearningTrader really is able to learn something useful. The implementation also provides a good playground for further experimentation for any interested member of the growing Quantopian community.

Improvement¶

There are several potential optimization targets in the QLearningTrader algorithm and the market environment.

The most obvious candidate for a redesign is the reward system, which assigns zero values to the close action, regardless of the wins or losses realised by closing a particular position. In combination with the often negative values for the alternative actions, this may be one of the reasons for the QLearningAgent's strong preference of short and noise-prone trades, despite the high gamma. A more general characteristic of the Q-table is the consistently low signal level, although it is less clear in what way this affects the overall performance of the agent. The system level could be increased by using a different reward function, such as (current_price-entry_price)position_size. A good approach for the close-reward problem could be the assignment of the previous position value to any close action.

Another promising optimization target could be the state space. The possible set of signal combinations derived from three indicators alone is already quite large and it gets much larger when additional indicators are used. A good approach to tackle the problem of state selection could be to identify a set of relatively strong indicator signals and then iterate over the power set to select the combination with the best results.

Finally, the set of free parameters of the model, specifically alpha,gamma,beta coud be optimzied by a grid search algorithm. This would however mostly help in finetuning the algorithm performance and should therefore be untertaken after the steps described above.